Mandela, looking frail and walking with a stick, told the rally: "Massive poverty and obscene inequality are such terrible scourges of our times ... that they have to rank alongside slavery and apartheid as social evils."
Wrapped in a black coat and brown fur hat to keep out the British winter chill, he made an impassioned speech in front of South Africa House, for decades a symbol of apartheid, and presented a group of overawed school children with "Make Poverty History" white wristbands.
"In this new century, millions of people in the world's poorest countries remain imprisoned, enslaved and in chains," he said. "They are trapped in the prison of poverty. It is time to set them free."
British Prime Minister Tony Blair has vowed to make Africa one of the priorities of his presidency of the Group of Seven (G7) industrialised nations this year.
Nobel peace laureate Mandela, who spent nearly 27 years in South Africa's apartheid-era jails, was freed in 1990 and elected the country's first black president four years later.
"I recently formally announced my retirement from public life and should not really be here," he said on Thursday. "However, as long as poverty, injustice and gross inequality persist in our world, none of us can truly rest."
He was speaking as part of the charity-driven "Make Poverty History" campaign.
He will take his message on Friday directly to the finance ministers when British Chancellor of the Exchequer Gordon Brown opens the meeting.
"In 2005, there is a unique opportunity for making an impact," Mandela said. "Tomorrow, here in London, the G7 finance ministers can make a significant beginning."
Mandela called for trade reforms, an end to the debt crisis that is costing poor countries 39 billion dollars a year and a boost in aid.